feat(datagrid): type expression index keys and read them back on SQLite, MySQL and DuckDB - #3080

Stacked on #3075

Summary

• An expression typed into an index's Columns cell, such as lower(email) or coalesce(a, b), is now an expression key on PostgreSQL, PGlite, SQLite, libSQL, Turso, Cloudflare D1, DuckDB and MySQL 8.0.13+. Before, the cell split it at every comma and read the pieces as column names, so the row was flagged and nothing could be saved.
• SQLite, libSQL, Cloudflare D1, MySQL and DuckDB now read expression key parts. Before, SQLite-family and MySQL reads dropped them and DuckDB labelled them as columns, so renaming such an index silently recreated a different one.
• SQLite-family partial indexes now show their condition, and MySQL DESC key parts survive a rename.
• On MySQL and MariaDB a changed index is replaced in one ALTER TABLE t DROP INDEX i, ADD ..., so a replacement the server rejects no longer loses the original.

Root cause

Authoring: StructureEditingSupport.updateIndex counted an entry as an expression only if the index already held it, and split everything else at every comma. It had neither the table's columns nor the engine's grammar to tell lower(v) from a name.

Round trip: only PostgreSQL read expressions. SQLite, libSQL and D1 joined pragma_index_info, whose name is NULL for an expression part (cid -2), and dropped it. MySQL dropped SHOW INDEX rows whose Column_name is NULL. DuckDB's regex split read (COALESCE(a, b)) as a column. Every one of those writers quoted each key part as an identifier, and none read DESC (MySQL) or the WHERE of a partial index (SQLite family).

Atomicity: SchemaStatementGenerator always split a modified index into a DROP and an ADD. MySQL DDL is not transactional, so a failing ADD left the table without the index.

What changed

• IndexKeyDialect (app, curated per database type like ForeignKeyDialect): takesPrefixLengths and takesExpressions. Unknown types get columns only.
• IndexKeyList (app, pure): splits the cell at depth-0 commas with SQLTokenCursor over the connection's execution grammar (SQLLexicalResolver.executionGrammar), so MySQL backslash strings, PostgreSQL E'' and dollar quotes, SQLite brackets and a session's NO_BACKSLASH_ESCAPES are all read the engine's way. A known column name (commas and apostrophes included) or an existing expression is matched whole first. Then each entry is classified: existing expression, known column (any case, with or without one pair of parentheses), quoted identifier, name(N) prefix where the engine takes prefixes, bare identifier, then balanced plain code as an expression where the engine takes them. A trailing ASC/DESC/NULLS FIRST|LAST, a comment, an unterminated literal or unbalanced text is left as a column so the missing-column check names it before Save. SQLTokenCursor gained a location accessor.
• StructureEditingSupport.updateIndex(_:at:with:keys:) takes an IndexKeyContext from both grid delegates (the inspector routes through them). indexModifiedIndices also tints Columns when expressions change. EditableIndexDefinition's catalog key shape now includes columnPrefixes, so a changed MySQL prefix retires the server's spelling.
• PluginKit (additive, pending kit 33, no Info.plist edits):
  • SQLIndexKeyList: reads a stored CREATE INDEX into its key list, key parts and predicate, strips a trailing sort order, unwraps one pair of parentheses, and reads a quoted identifier, all on the kit's SQLFeatureLexer.
  • SQLiteIndexCatalog: one per-table and one schema-wide query (pragma_index_list + pragma_index_xinfo ... key = 1 + sqlite_master.sql), grouping into PluginIndexInfo with expressions, whereClause and the verbatim key list as ddlMethodAndKeys, plus the writer (createStatement, keyList). SQLite, libSQL and D1 all use it; their three copies of the read and writers are gone.
  • PluginDatabaseDriver.generateModifyIndexSQL(table:oldIndexName:newIndex:), defaulting to nil.
• SchemaStatementGenerator keeps a modified index whole when the driver answers that hook and the save changes no column; otherwise it splits as before (the halves belong on opposite sides of column work).
• MySQL: MySQLIndexRow carries a key part (column with prefix, or expression) and whether it is descending. The reads take Expression from SHOW INDEX by column name and STATISTICS.EXPRESSION only on MySQL 8.0.13+ (NULL elsewhere, so MariaDB never errors), with one level of backslash escaping removed. Grouping fills expressions and, when a part is DESC, the key spelling. The writer puts expressions in parentheses. schemaOperationRefusal(.addIndex) refuses expression keys on MySQL below 8.0.13 and on MariaDB. generateModifyIndexSQL answers for MySQL and MariaDB only.
• DuckDB: DuckDBIndexClauses reads duckdb_indexes().sql through SQLIndexKeyList (a parenthesized part is an expression, a quoted name is unquoted) and writes expressions as (expr). extractIndexColumns and its regex are gone.
• CockroachDB stays columns-only for typed expressions (see below).
• Docs: features/table-structure.mdx (an Expression keys section replaces "An expression typed here is read as a column name") and databases/mysql.mdx (8.0.13 and MariaDB limitation). CHANGELOG: one Added, four Fixed.

Measured

• PostgreSQL 17.11: the writer's CREATE INDEX "ix" ON ... USING btree ("tenant_id", (lower(email))), ((a || ', ' || b), "id") and a multi-line (CASE ... END) all ran; ((lower(email) DESC)) is syntax error at or near "DESC".
• MySQL 8.4.11: SHOW INDEX and STATISTICS report Column_name NULL with Expression lower(`v`) and Collation D for ((lower(v)) DESC). STATISTICS.EXPRESSION adds one backslash level (_utf8mb4\'it\\\'s\'); removing it gives the SHOW CREATE TABLE form, and replaying that created an identical index. Every statement the writer tests expect ran: ALTER TABLE `t` DROP INDEX `i_fn`, ADD INDEX `i_fn_lower` ((lower(`v`)) DESC) USING BTREE re-read with Collation D, (`v` DESC, `id`) likewise, and (`id`, (coalesce(a, b)), `email`(20)) kept its prefix. A combined replace whose ADD fails (Unknown column 'zz' in 'functional index', and a DESC inside the parentheses, 1064) left the index in place, where the split form lost it.
• MariaDB 13.0.2: the combined DROP INDEX, ADD INDEX works; ADD INDEX i ((lower(v))) is 1064; STATISTICS has no EXPRESSION column and SHOW INDEX has none either.
• SQLite 3.54.0: pragma_index_xinfo reports cid -2 with a NULL name for expression parts and desc/coll per part; sqlite_master.sql keeps the statement verbatim (((lower(v))) stays doubled). The catalog query was run live in the new tests against the system SQLite.
• DuckDB 1.5.4: sql stores every expression key wrapped in one more pair of parentheses (lower(v) is stored as the key (lower(v)), (a || ', ' || b) as (((a || ', ') || b))), stores (v) as the column v, accepts v DESC but drops the DESC, rejects (lower(v) DESC), and has no partial indexes.

Tests

Run through verify.sh on this machine, every suite confirmed executed:

• New and directly changed, 216 cases, all passed: IndexKeyListTests 16, StructureEditingSupportIndexKeyTests 9, StructureChangeManagerIndexExpressionTests 5, StructureEditingSupportFieldDiffTests 13, StructureEditingSupportBooleanParsingTests 60, StructureIndexTypeMenuTests 4, IndexDefinitionCatalogSpellingTests 12, SchemaStatementGeneratorPluginTests 25, MySQLIndexGroupingTests 7, MySQLIndexKeyWriterTests 6, MySQLFunctionalKeyPartsTests 6, MySQLCreateTableTests 10, SQLiteIndexCatalogTests 8, SQLiteCreateTableDDLTests 9, DuckDBIndexClausesTests 5, SQLIndexKeyListTests 6, SQLTokenCursorTests 16.
• Other suites that own a changed type (EditableIndexDefinition, SchemaStatementGenerator, the grid delegates, SQLTokenCursor), 248 cases, all passed: StructureGridDelegateInspectorTests, StructureGridDelegateAddRowTests, CheckConstraintStatementTests, SchemaOperationRefusalTests, ClickHouseIndexEditTests, StructureChangeManagerClusteredIndexTests, StructureChangeManagerCatalogSpellingTests, CrossEngineIndexExpressionTests, CrossEngineIndexTypeTests, CrossEngineKeyTranslationTests, SchemaSyncScriptBuilderTests, StructureDiffEngineTests, TableStructureIndexReplayTests, CreateTableDraftBuilderIndexExpressionTests, CreateTableDraftBuilderTests, IndexDefinitionTests, IndexDefinitionPasteTests, IndexTypeTests, SchemaChangeTests, StructureChangeGuardTests, SQLSetAssignmentsTests, PluginIndexMappingCoverageTests, StructureRowProviderTests.
• Builds: TablePro, SQLiteDriver, LibSQLDriverPlugin, CloudflareD1DriverPlugin, MySQLDriver, DuckDBDriver, PostgreSQLDriver.
• check-pluginkit-abi.sh against the merge base: additions only (generateModifyIndexSQL with its default, SQLIndexKeyList, SQLiteIndexCatalog); the one changed line is fix(plugin-postgresql): follow pg_dump's index rule so invalid indexes stay out of dumps and copies #3075's @_disfavoredOverload on an unchanged PluginIndexInfo init. Kit 33 is already pending, so no bump and no Info.plist edits.
• SwiftLint --strict on all 41 changed Swift files: two violations, both on untouched older lines (SQLiteCreateTableDDLTests.swift:8 import order, SchemaStatementGeneratorPluginTests.swift:543 unused closure parameter). Docs checks pass.

What turns each red without this change:

• IndexKeyListTests, StructureEditingSupportIndexKeyTests.typedExpressionIsAnExpression, StructureChangeManagerIndexExpressionTests.typedExpressionOnANewIndex: the old updateIndex splits coalesce(a, c) into coalesce(a and c), and the change manager reports Index references a column that does not exist: lower(email).
• SchemaStatementGeneratorPluginTests.modifyIndexInOneStatement: remove the keepsIndexModifiesWhole arm in sortByDependency and the modify is split into DROP + CREATE.
• MySQLIndexGroupingTests.functionalKeyPartsAreRead / catalogEscapingIsRemoved / descendingKeysAreSpelled, MySQLIndexKeyWriterTests.renameKeeps*: the old MySQLIndexRow required a column (the rows were dropped) and carried no collation, so the rename wrote no DESC.
• MySQLFunctionalKeyPartsTests: new gate; returning nil from refusal fails the MySQL 8.0.12 and MariaDB cases.
• SQLiteIndexCatalogTests: the old pragma_index_info read gives i_mix as [id], i_fn as [] and no condition; the old writer quotes a || ', ' || b as an identifier.
• DuckDBIndexClausesTests: the old regex split read (COALESCE(a, b)) as a column and quoted it on write.
• IndexDefinitionCatalogSpellingTests.prefixEditRetiresTheKeySpelling: without columnPrefixes in the key shape a changed prefix keeps writing the old spelling.
• StructureEditingSupportFieldDiffTests.indexExpressionsChanged: without the expressions comparison nothing is tinted.

Not added: the design's StructureIndexExpressionUITests. The flow is deterministic, but lanes run in parallel on one shared screen, so an XCUITest could not be run here, and an unrun UI test would put a red gate on CI. The unit suites cover cell text to staged change to SQL, and SQLiteIndexCatalogTests runs the write and the re-read against real SQLite.

Before / After

Screenshots to be added. States to capture, on the sample SQLite database and a PostgreSQL table:

1. Indexes tab, new index row, Columns typed lower(name): before, the row is flagged Index references a column that does not exist: lower(name); after, no flag and Save enabled.
2. After Save and reload, the row reads lower(name) in Columns.
3. SQLite partial index: before, Condition empty; after, it shows its WHERE predicate.
4. MySQL 8.4 table with KEY i_fn ((lower(v)) DESC): before, no i_fn row at all; after, a row reading lower(`v`).

Critique points not taken

• CockroachDB: taken in the second form the critique offered. It stays columns-only for typed expressions rather than fixing CockroachPluginDriver.fetchIndexes, because another lane is changing that driver for materialized views; the read fix is listed below.
• iOS: listed as follow-ups rather than changed here (below).
• IndexKeyDialect keeps exactly takesPrefixLengths and takesExpressions. The typed sort-order refusal is uniform across engines instead of a third flag: the grid model has no per-key sort order, so a typed DESC is refused everywhere, and an existing one is kept through the server's spelling.
• TiDB and OceanBase are not refused by the driver (the design refused them). They are unmeasured, TiDB reports expression indexes of its own through SHOW INDEX, and refusing would block renaming one. Typed expressions are still not offered on those types, and the one-statement modify is not used for them (TiDB before 6.2 rejects multi-change ALTER TABLE), so their modify stays as before.

Deliberately not fixed here

• CockroachDB SHOW INDEXES reports an expression part as the hidden column crdb_internal_idx_expr and keeps storing rows as key parts: Plugins/PostgreSQLDriverPlugin/CockroachPluginDriver.swift:115. Read definition for those rows and skip storing = t, then add CockroachDB to IndexKeyDialect.
• iOS SQLite reads indexes through pragma_index_info and drops expression parts: TableProMobile/TableProMobile/Drivers/SQLiteDriver.swift:186. It can call SQLiteIndexCatalog directly.
• iOS MySQL drops SHOW INDEX rows with a NULL Column_name: TableProMobile/TableProMobile/Drivers/MySQLDriver.swift:212.
• TiDB and OceanBase: STATISTICS.EXPRESSION is only selected on MySQL 8.0.13+, so the schema-wide read (Compare) on those servers still drops functional key parts: Plugins/MySQLDriverPlugin/MySQLPluginDriver+BulkMetadata.swift:44. Needs a measured gate per engine.
• Under NO_BACKSLASH_ESCAPES a MySQL expression read from the catalog is written back in its backslash-escaped form, which that session reads differently. Rare, and the same for every other catalog text the MySQL writer replays.
• libSQL, Turso and Cloudflare D1 are unmeasured here (no accounts); they read through the same pragma_index_xinfo and sqlite_master join SQLite runs, and D1 already reads sqlite_master for triggers. libSQL and D1 are registry plugins and need a re-release against kit 33.